Which issue does this PR close?

Partially addresses #3510. Closes #4014. Related to #121. Closes #242.

Rationale for this change

CometNativeScanExec (the native DataFusion-based Parquet V1 scan) falls back to Spark for queries that use Dynamic Partition Pruning (DPP), even though the non-AQE DPP path works through the existing lazy partition serialization infrastructure from #3511 and #3349.

Spark has two DPP paths:

The prior gate rejected both. This PR narrows it to only reject AQE DPP.

What changes are included in this PR?

DPP gate changes

• Added isAqeDynamicPruningFilter to distinguish AQE from non-AQE DPP by checking for SubqueryAdaptiveBroadcastExec inside InSubqueryExec.
• CometScanRule.transformV1Scan() and CometNativeScan.isSupported() now reject only AQE DPP.
• Removed spark.comet.dppFallback.enabled config — redundant since isSupported() unconditionally rejects AQE DPP.

DPP subquery resolution

• CometNativeScanExec.serializedPartitionData explicitly resolves DPP subqueries via updateResult() before accessing file partitions. This is needed because serializedPartitionData can be triggered from findAllPlanData on a BroadcastExchangeExec thread, outside the normal prepare() -> executeSubqueries() flow.
• CometNativeScanExec.outputPartitioning uses perPartitionData.length instead of originalPlan.inputRDD.getNumPartitions to avoid triggering FileSourceScanExec codegen on unresolved DPP expressions.
• CometNativeScan.convert() uses scan.selectedPartitions (static) instead of scan.getFilePartitions() for object store option extraction, since DPP subqueries aren't resolved at planning time.

Broadcast exchange reuse (CometSubqueryBroadcastExec)

ReuseExchangeAndSubquery runs after Comet rules. Without intervention, the join side has CometBroadcastExchangeExec and the DPP subquery has BroadcastExchangeExec — different types, no reuse, double broadcast.

CometSubqueryBroadcastExec replaces SubqueryBroadcastExec in DPP expressions. It wraps a CometBroadcastExchangeExec and decodes Arrow broadcast data to extract DPP key values. CometExecRule.convertSubqueryBroadcasts() handles the conversion during transformUp, stripping the CometNativeColumnarToRowExec transition that ApplyColumnarRulesAndInsertTransitions inserted for the original BroadcastExchangeExec. Both sides now have CometBroadcastExchangeExec, so ReuseExchangeAndSubquery matches them.

Shuffle fallback

stageContainsDPPScan in CometShuffleExchangeExec checks for FileSourceScanExec with DPP. Scans converted to CometNativeScanExec don't match, so the shuffle proceeds natively. The COMET_DPP_FALLBACK_ENABLED config gate was removed; the check now always runs.

How are these changes tested?

• "non-AQE DPP with BHJ works with CometNativeScanExec" — verifies correct results, CometNativeScanExec in plan, DynamicPruningExpression in partition filters
• "non-AQE DPP with SMJ works with CometNativeScanExec" — same for sort-merge join path
• "non-AQE DPP with BHJ reuses broadcast exchange" — verifies CometSubqueryBroadcastExec, ReusedExchangeExec, and exactly 1 CometBroadcastExchangeExec
• Existing "DPP fallback" and "DPP fallback avoids inefficient Comet shuffle" tests updated for new message text and config removal
• CometDppFallbackRepro3949Suite and CometShuffleFallbackStickinessSuite pass with config references removed
• TPCDS golden plans regenerated
• Spark SQL test diffs updated for 3.4.3, 3.5.8, and 4.0.1:
  • CometNativeScanExec added to collectDynamicPruningExpressions in DynamicPartitionPruningSuite and DynamicPartitionPruningHiveScanSuite
  • CometSubqueryBroadcastExec added to checkPartitionPruningPredicate, checkDistinctSubqueries, countSubqueryBroadcasts, countReusedSubqueryBroadcasts, and SPARK-38674 test
  • CometNativeScanExec / CometNativeColumnarToRowExec added to SubquerySuite SPARK-26893 test
  • RemoveRedundantProjectsSuite "join with ordering requirement" unignored (exchange reuse now works)

TPC-DS native acceleration improvement (Spark 3.5, v1.4)

78 queries improved. Previously DPP queries fell back entirely; now they run natively.